Quicker Q-Learning in Multi-Agent Systems

Multi-agent learning in Markov Decisions ProbK i s chanenging because of the presence ot two credit assignment problems: 1) How to credit an action taken at time step t for rewards received at t’ > t ; and 2 ) How to credit an action taken by agent z considering the system reward is a function of the actions of all the agents. The first credit assignment problem is typically addressed with temporal difference methods scch as Q-learning OK TD(X) The second credit assi,onment problem is typically addressed either by hand-crafting reward functions that assign proper credit to an agent, or by making certain independence assumptions about an agent’s state-space and reward function. To address both credit assignment problems simultaneously, we propose the “Q Updates with Immediate Counterfactual Rewards-learning” (QUICRlearning) designed to improve both the convergence properties and performance of Q-learning in large mulh-agent problems. Instead of assumng that an agent’s value function can be made independent of other agents, this method suppresses the impact of other agents using counterfactual rewards. Results on multl-agent grid-world problems over multiple topologies show that QUICR-learning can achieve up to thirty fold improvements in performance over both conventlonal and local Q-learning in the largest tested systems. -_ __ _ _